scholar-api

An unofficial, open-source API for searching Google Scholar — and for checking whether a paper's references are real.

Google Scholar has no official API. The alternatives are commercial or heavyweight; this one is small, readable, and installs in one command.

Point it at a submission PDF and it tells you which of its references are real.

Install

Not published to PyPI yet, so install from a clone. Once it is released, pip install "scholar-api[all]" will be the whole story.

git clone https://github.com/SamisLife/scholar-api
cd scholar-api

python -m venv .venv
.venv\Scripts\activate            # Windows
# source .venv/bin/activate       # macOS / Linux

pip install -e ".[all]"           # PDF support + the HTTP service

Everything below assumes that virtualenv is active. If you would rather not activate it, put .venv\Scripts\ in front of each command — running python or pip from another shell uses your system Python, which does not have this package installed.

Check a paper

scholar verify submission.pdf
scholar verify references.bib

37 references checked

  OK   VERIFIED       29   matched with high confidence
  ??   UNCERTAIN       6   partial match - check these yourself
  --   UNVERIFIABLE    0   repos, URLs - expected, not suspicious
  MISS NOT_FOUND       2   no source has any record of these

MISS  NOT_FOUND
      Petrov, A. and Sundaram, K. Differentially Private Federated Attestation
      over Untrusted Enclaves. 31st USENIX Security Symposium, 2022.
      - The DOI given in the reference (10.5555/3488021.3488107) is not
        registered. Invented identifiers are a common feature of fabricated
        citations.

Accepts .pdf, .bib, and plain-text reference lists. Exit code 0 if nothing was flagged, 5 if something was — so it can gate a CI job.

Search

scholar search "attention is all you need"
scholar search "side channel" --author Kocher --from 2015 --pages 2
scholar search "privacy" --in "USENIX Security" --phrase "differential privacy"
scholar cited-by 2960712678066186980
scholar versions 2960712678066186980
scholar bibtex 5Gohgn6QFikJ

from scholar_api import Scholar, SearchQuery

with Scholar() as scholar:
    for paper in scholar.search(SearchQuery(text="side channel", author="Kocher")):
        print(paper.year, paper.title, paper.cited_by_count)

Every field except the title can be None — Scholar genuinely omits years, citation counts and links depending on the result.

Check a reference

scholar lookup "Chen, L. and Kumar, R. Neural Manifold Regularization for \
Adversarial Robustness. In Proc. USENIX Security, 2021."

??    UNCERTAIN   (confidence 0.49)

  best match:
      Adversarial robustness of beyond neural network models
      Pin-Yu Chen, Cho-Jui Hsieh
      2023 | Adversarial Robustness for Machine Learning

  - Best candidate's title differs from the one cited.

  sources: crossref, scholar (18 candidates)

Retracted work is flagged separately

A reference can be entirely real and still be one a reviewer must catch. Citing retracted work is something program committees ask about, and it is the one finding that hides inside VERIFIED -- the citation is genuine, so nothing else about it looks wrong.

Every confirmed match is checked against Crossref's retraction and correction notices, from the record already fetched: no extra request, no extra dependency. Retractions, partial retractions, withdrawals and expressions of concern are reported and lead the results. Corrections and errata are not -- a corrigendum usually means a figure was relabelled, and treating that like a retraction would train a reviewer to ignore the warnings.

Retraction is reported alongside the verdict rather than as a fifth one: the four verdicts answer "does this exist", and a retraction answers "should it be cited". They are different questions about the same reference.

Four verdicts

The distinction between the last two matters:

Verdict	Meaning
VERIFIED	A matching record was found and corroborated.
UNCERTAIN	Something similar exists, but not close enough to confirm. Look yourself.
UNVERIFIABLE	Cites a repository or web page. Not suspicious — no index holds these.
NOT_FOUND	Every source answered, and none has any record of this work.

UNVERIFIABLE exists because papers legitimately cite repositories, blog posts and vendor pages. Reporting those as missing would bury the one fabricated reference among false alarms and train a reviewer to ignore the tool.

CVEs are checked, not excused. A reference citing CVE-2013-6117 is resolved against MITRE's register: a published identifier is confirmed and its official description is shown, an identifier that was never issued is NOT_FOUND. The shared-year shorthand (CVE-2021-33044/33045) is expanded, so both are checked. Security bibliographies are full of CVEs, and leaving them unverified would give a fabricated citation the one place nobody looks.

Exit codes: 0 verified, 5 not verified, 3 blocked, 4 parse failure.

HTTP service

With the virtualenv active:

uvicorn scholar_api.service:app

Or without activating it — note it is the venv's Python, not the system one:

.venv\Scripts\python.exe -m uvicorn scholar_api.service:app

docker build -t scholar-api .
docker run -p 8000:8000 -v scholar-cache:/cache scholar-api

Then open http://127.0.0.1:8000 — paste a reference list or drop in a PDF and watch the verdicts stream in. http://127.0.0.1:8000/docs is the API browser.

Every result carries somewhere to go next. A confirmed reference links straight to the record — the DOI, the cited page, or the CVE register entry. Anything unconfirmed gets a search that is already typed out, Google Scholar and Google, built from the reference's title rather than the whole citation. Those are the ones a reviewer has to settle by hand, and retyping a title is exactly the friction that gets a tool abandoned. The links are in the API too, as links on each assessment.

Interactive docs at /docs describe every route with worked examples and the error contract. GET / is the web interface. GET /search, /paper/{cluster_id}/citations, /paper/{cluster_id}/versions, /paper/{result_id}/bibtex, /verify, /verdicts, /health, and /api for a machine-readable index. POST /verify/text and POST /verify/file back the page, streaming newline-delimited JSON so results appear as they are checked rather than after several minutes.

curl "http://127.0.0.1:8000/search?q=attention+is+all+you+need"
curl -G http://127.0.0.1:8000/verify --data-urlencode \
  "reference=Lamport, L. Time, clocks. CACM, 1978."

Design notes

It never turns a failure into an empty result. This is the constraint the rest of the code is arranged around. A CAPTCHA raises BlockedError, a markup change raises ParseError, a throttled index raises TransportError, and a reference whose sources could not all be reached comes back UNCERTAIN rather than NOT_FOUND. In a tool that tells a reviewer whether a citation is real, "no results" and "we were blocked" must never be confused.

Provider relevance scores never decide anything. Crossref returns a confidently-ranked real paper for references that do not exist — an invented USENIX citation scores 39.8 against 70.8 and 85.4 for genuine ones. Scores rank candidates; the verdict comes from an explicit comparison of title, authors and year that the tool can explain back to you.

Index years are not trustworthy. Both Crossref and OpenAlex date "Attention Is All You Need" to 2025, because recent preprint reposts outrank the 2017 original. Year disagreement therefore cannot veto a match on its own, and Google Scholar is consulted when the open indexes cannot date a work.

It caches, and that matters. Responses go to SQLite (scholar cache path). Repeating a check costs nothing and makes no request. Raw HTML is stored rather than parsed results, so a parser fix applies retroactively and a result you reported last month can be reproduced today.

Where the time goes. Almost all of it is network, deliberately paced. The work around it is cheap: text normalisation is memoised, so matching a reference against eighteen candidates costs 0.4 ms rather than 5 ms, and the cache runs in WAL mode because checking a bibliography is a long run of small writes. Re-running a 37-reference paper against a warm cache takes about two seconds, most of which is Python starting up.

It is slow on purpose. Requests to Scholar are spaced out and jittered, with exponential backoff; requests to Crossref and OpenAlex are paced too. You can tune the delay (--delay); you cannot remove it.

Configuration

Variable	Purpose
SCHOLAR_CACHE_DIR	Where the response cache lives.
SCHOLAR_API_MAILTO	Your email. Puts you in Crossref's and OpenAlex's faster "polite pool". Recommended for bulk checking; never sent unless you set it.
HTTPS_PROXY / HTTP_PROXY	Standard proxy variables, honoured automatically. Use your institution's proxy or your own VPN if you have one.

If you get rate limited

Every source here throttles — Crossref, OpenAlex, DBLP and Scholar all do, and there is no endpoint you can hammer. What actually helps:

1. Set SCHOLAR_API_MAILTO. Free, and it moves you into Crossref's and OpenAlex's faster pools.
2. Keep the cache. Re-checking a paper costs nothing: a 37-reference paper takes 39 seconds cold and 2 seconds warm, with identical verdicts. Don't clear it, and point SCHOLAR_CACHE_DIR at a shared location if a group of reviewers is checking overlapping bibliographies.
3. Use --no-scholar for a first pass. Crossref and OpenAlex have far more generous limits. Re-run without the flag only for the references they could not settle.
4. Wait it out. When a source reports a quota-length backoff, the tool records it and stops asking for the rest of the run — one request instead of thirty — and says so in the output rather than pretending the references were checked.

What this project deliberately does not do is rotate through proxies to get around those limits. Aside from being the kind of evasion that would make the tool indefensible to the venues it is meant to serve, free proxy pools would route your colleagues' unpublished submissions through unknown third parties. If you have a proxy of your own, the standard environment variables above work.

Development

pytest                           # offline, 260 tests against recorded responses
pytest -m live                   # opt-in; queries Scholar, Crossref, OpenAlex, arXiv
ruff check .
python tools/fetch_fixtures.py   # regenerate HTML fixtures after a layout change
python tools/make_pdf_fixture.py # rebuild the synthetic PDF fixture

Every CSS selector lives in parse.py; see PARSING.md. The reference corpus that defines "working" for verification is tests/corpus.py — real references, invented ones, and real works cited wrongly. See CONTRIBUTING.md.

Known limits

PDF extraction expects numbered bibliographies ([1], 1.), which is what the security and systems venues use. Author-year styles are split on blank lines instead, and the tool says so rather than pretending the count is reliable. Scanned PDFs need OCR first; they are reported, not silently skipped.

Grey literature is flagged more often than it should be. Papers cite vendor documentation, news articles and press releases that no index holds and that carry no URL to detect them by. Those come back UNCERTAIN or NOT_FOUND with a note explaining that absence is weak evidence for that kind of citation. The alternative — excusing anything that looks unacademic — would let a fabricated reference through, which is the worse error.

Run without --no-scholar if you can. On a real 37-reference paper, the open indexes alone confirmed 19; adding Scholar took it to 29. Scholar carries the theses, workshop papers and older non-DOI work the others miss.

Legal

Scraping Google Scholar is against Google's Terms of Service, as it is for every tool in this category. This project stays on the polite side of the line: aggressive caching, a low default request rate, no attempt to bypass authentication, and no CAPTCHA solving. Crossref and OpenAlex are official APIs used as documented. If that is not acceptable for your purposes, don't use it — or run with --no-scholar and never touch Scholar at all.

License

MIT